[fix](paimon) Preserve statement schemas and timestamp precision - #67904
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
There was a problem hiding this comment.
Request changes: four findings remain (three P1, one P2). The NTZ constructor change itself is sound, but the schema/data-generation handling is not end-to-end safe.
Critical checkpoint conclusions:
- Goal and proof: The goal is to retain the current Paimon schema while fencing data and to preserve NTZ sub-millisecond predicates. M1, M2, and M4 show the schema/MVCC goal is not achieved in branch, cached-table, and schema-derived system paths; M3 shows the end-to-end precision regression does not exercise NTZ.
- Scope and focus: The patch is otherwise focused across three production files and targeted tests. No additional user focus was supplied, so the full changed-file scope was reviewed.
- Concurrency: No threads or locks are added. External data/schema commits can race statement binding and planning; the unfenced branch transition is M1 and exact-schema drift is M2.
- Lifecycle and static initialization: Table handles, transient Paimon tables, and scan-option copies are statement-scoped; no new static initialization or resource-release issue was found. The branch handle reload lifecycle is part of M1.
- Configuration: No production configuration is added. The regression restores force_jni_scanner in finally.
- Compatibility: No public SPI, thrift, storage format, symbol, or rolling-upgrade surface changes. Paimon 1.3.1 provides the APIs used.
- Parallel paths: Normal, branch, options, native, JNI, system-table, privilege, fallback, partition, and statistics paths were traced. Distinct gaps are M1, M2, and M4.
- Special conditions: The new FileStoreTable/preserve-marker condition is documented but insufficient: it excludes schema-derived wrappers (M4), and the marker carries no exact schema generation (M2).
- Test coverage: Added unit cases cover a fresh ordinary table and the converter helper, but miss branch mutation, warm catalog cache, schema-derived system aliases, and a true NTZ end-to-end setup (M1-M4). No additional negative/empty-branch test covers the first-commit race.
- Test results: Expected rows are deterministic for the stable setup, but the precision assertions can pass via residual filtering because the Spark column is LTZ (M3). No result file is changed.
- Observability: No new logging or metrics are required for this local read-path change; no separate observability defect was found.
- Transactions and persistence: No Doris transaction, EditLog, failover, or persisted-metadata path is modified.
- Data writes and atomicity: Production changes are read-only; no write/crash atomicity path is modified.
- FE/BE variables: No new cross-process variable or thrift field is introduced. Existing serialized-table/native paths are inconsistent only as described in M2/M4.
- Performance: The added map/table copies are bounded per statement; no distinct CPU, memory, or asymptotic regression was found.
- Other issues: No additional correctness, error-handling, security, or style blocker remained after three converged review rounds.
Static review only: per the review contract, no builds or tests were run, so author/CI test claims were not independently executed.
TPC-H: Total hot run time: 16965 ms |
TPC-DS: Total hot run time: 82255 ms |
ClickBench: Total hot run time: 14.79 s |
### What problem does this PR solve? Related PR: apache#67904 Problem Summary: Cached Paimon tables and system wrappers could use a different schema than statement binding, and branch reads could observe later commits. Carry the exact schema generation and positive or empty branch data fence through binding, native planning and JNI serialization. Use explicit NTZ literals in the timestamp precision regression. ### Release note Paimon statement reads retain consistent schema and data generations. ### Check List (For Author) - Test: Paimon package build (570 passed, 1 skipped); FE MVCC unit tests (68 passed); FE Checkstyle. External regression updated, not run locally. - Behavior changed: Yes, schema and data fences remain consistent through reads. - Does this need documentation: No.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of exact head 4ffdc1fc1d5b4dbf94ad6dac896b9c24a9976bb3 against base 73b2a153cb5eb34b474386d678ee4a02af5788bd. I am requesting changes for four P1 correctness defects and one P2 regression-standard defect, detailed inline.
Checkpoint conclusions:
- Goal and proof: the ordinary stale-table, system-wrapper, branch-fence, and microsecond cases are addressed, but schema-owned options, fallback topology, TIMESTAMP(7..9), and Iceberg's generic latest pin remain inconsistent; the release-note claim is therefore not yet established for supported paths.
- Focus and scope: all 12 changed files plus relevant callers and consumers were reviewed; no user-specific focus was supplied. The change is otherwise focused.
- Concurrency and lifecycle: the schema memo is statement-scoped, concurrently safe, and discarded with metadata; handle identity separates base/system/branch tables. No lock-order or teardown issue found.
- Configuration and compatibility: no new user configuration, persisted format, public symbol, storage protocol, or FE-BE field is introduced. The internal marker is stripped before Paimon/backend options.
- Parallel paths and conditions: plain, OPTIONS, branch, empty, system-wrapper, fallback/decorator, native dictionary, catalog-loader removal, JNI serialization, and Iceberg paths were traced. The surviving defects are the five inline points; prior GitHub threads were treated as duplicate fences.
- Tests and results: the changed unit tests statically reach the main intended paths but miss renamed schema-owned options, stale fallback pairs, precision above 6, and mixed Iceberg cache generations. Stable regression rows also bypass the required
qt_/order_qt_generated-output workflow. This review ran no builds or tests. The author reports 570 Paimon tests, 68 MVCC tests, and Checkstyle passing; the external Spark/Doris regression was not run locally. - Observability and performance: existing errors/logging are adequate, the per-statement map and schema lookup are bounded, and no separate observability or performance blocker was found.
- Transactions, persistence, and data writes are unchanged. Memory-safety and BE nullable checkpoints are not applicable to this FE Java/test-only diff.
TPC-H: Total hot run time: 17215 ms |
TPC-DS: Total hot run time: 83453 ms |
ClickBench: Total hot run time: 14.97 s |
### What problem does this PR solve? Related PR: apache#67904 Problem Summary: Restoring a bound Paimon schema could retain field-referencing options from a different generation or overwrite a fallback branch's schema and identity. Exact nanosecond timestamp pushdown could reject values that compare equal after Doris truncation. Iceberg cached schema pins could expose live partition names and specs. Preserve each schema's options and fallback provenance, retain privilege checks, keep high-precision comparisons residual, and pin Iceberg partition specs with their schema. ### Release note External scans retain consistent schema and partition metadata. Paimon nanosecond timestamps are compared at Doris precision without unsafe source pruning. ### Check List (For Author) - Test: 1,963 connector unit tests passed, 6 skipped; one existing Iceberg test failure reproduced before the changes and excluded. FE Checkstyle and plugin packages passed. Generated and verified all five regression baselines through the regression harness. - Behavior changed: Yes, preserve schema provenance and avoid false timestamp pruning. - Does this need documentation: No.
|
run buildall |
|
/review |
|
Addressed the five review findings in 75ceac1:
Validation: 1,963 connector tests passed and 6 skipped; one pre-existing Iceberg MERGE test failure was reproduced on the previous PR head and excluded. FE Checkstyle, FE build, and plugin packages passed. Local regression used the current FE/JNI extension with an existing BE and disabled the FE local-shuffle planner only on that isolated test instance for compatibility. The PR description has been updated. GitHub review-reply and thread-resolution APIs are returning server errors, so this top-level comment records the fixes while those discussion updates remain unavailable. |
TPC-H: Total hot run time: 16922 ms |
There was a problem hiding this comment.
Static review outcome: request changes on exact head 75ceac12b856aca84f572e2337b52870837950b7.
Goal and focus: this PR separates data, schema, and Iceberg spec generations for statement-stable external-table reads and fixes Paimon timestamp literal precision. No additional review focus was supplied. The implementation addresses the previously reported branch, wrapper, bound-schema, schema-builder, timestamp-test, and regression-baseline gaps, but three statement-consistency blockers remain.
Blocking findings:
- Paimon exact-schema discovery/materialization/restoration performs schema-file I/O after the connector authentication/TCCL scope has closed.
- Iceberg records the latest spec ID for schema assembly but drops it before partition-view construction and cache identity.
- The generic MVCC layer computes an exact local pinned schema, then builds partition items through ambient no-argument schema accessors before the pin is registered.
Checkpoint conclusions:
- Goal alignment and tests: the changed tests exercise warm-table Paimon schema restoration, branch/options/native/JNI paths, NTZ precision, Iceberg schema assembly, and generated regression output. They stop before the three failing boundaries described inline.
- Scope, clarity, and reuse: the changes are focused and reuse the statement snapshot abstraction, but the coordinate is not propagated through every generic/connector consumer.
- Concurrency and synchronization: statement maps and immutable/single-flight caches show no new lock-order or mutation race; omitting
specIdfrom the derived cache identity is nevertheless a cross-generation correctness error. - Lifecycle and cleanup: statement metadata ownership and connector close paths are sound. Snapshot registration timing leaves partition construction unable to observe the local pin.
- Configuration: no new configuration contract is introduced. Existing metadata-cache behavior and secured connector deployments expose the reported gaps.
- Compatibility and parallel paths: no persisted or FE/BE wire format changes were found. Base tables, OPTIONS/system/fallback/branch variants, empty snapshots, native/JNI reads, and RANGE/LIST Iceberg paths were checked; existing comments remain duplicate fences.
- Conditions and error handling: timestamp comparison, IN/OR, null propagation, and residual filtering are consistent. The remaining schema I/O can escape authentication, while mixed partition arity/types can fail analysis or silently omit partitions.
- Test execution: per the review contract, I did not build or run tests; conclusions are static. Changed tests were inspected, including the generated ordered regression baseline.
- Observability: no separate logging or metrics gap was identified; fixing context and generation propagation is preferable to diagnosing the resulting failures later.
- Persistence, transactions, and writes: not applicable beyond read-side metadata coordinates; no EditLog, storage-format, or data-write change is present.
- Resource and null safety: Java-only changes introduce no BE ownership or memory-safety issue; nullable timestamp/filter paths retain residual evaluation.
- Performance: statement memoization and bounded caches are reasonable. The spec-view cache key must include the complete generation coordinate.
- Security and other risks: the Doris threat model was applied. Privilege-wrapper retention is sound and no security vulnerability is asserted; the Paimon issue is authenticated connector correctness/availability in secured deployments.
The three inline P1 findings must be fixed and covered before approval.
TPC-DS: Total hot run time: 83385 ms |
ClickBench: Total hot run time: 15.27 s |
|
/review |
924060929
left a comment
There was a problem hiding this comment.
Re-reviewed the current head with the local pipeline-style review. The previously reported schema, timestamp, authentication, and Iceberg identity fixes are present, but four distinct Major regressions remain: captured-schema lifetime across INSERT replanning, fallback-child schema restoration, catalog option provenance, and HMS Iceberg capability forwarding. Details are inline.
There was a problem hiding this comment.
Request changes. Static review of exact head 54a5294b6cb8490693a157d0d465a47e5e15c9a6 found two additional P1 correctness/availability failures. An equivalent P2 catalog-option provenance issue was posted meanwhile in discussion r4004077352, so I have not duplicated it.
Critical checkpoints
- Goal and proof: The PR aims to keep external-table data, schema, spec, and predicate semantics coherent across schema-only evolution and recreation. The direct paths are substantially covered, but the goal is incomplete: Paimon can cross physical generations within one statement, and valid UUID-less Iceberg evolution is rejected.
- Focus/minimality: The 19-file diff is focused on Paimon/Iceberg MVCC pinning, generic FE consumers, and their tests; no unrelated edits were found.
- Concurrency/thread safety: External catalog ALTER/commit/DROP/CREATE operations are the relevant concurrent actors. Cache values are immutable and statement maps use the existing concurrent machinery; no new lock-order or deadlock issue was found. The Paimon logical handle key nevertheless lacks a physical-generation fence, producing the first inline issue.
- Lifecycle: Statement metadata, independently resolved Paimon handles, Iceberg statement-frozen tables, cache invalidation/retry, leases, authentication scopes, and decorator teardown were traced. The two inline issues are lifecycle/cache-boundary defects; no separate leak, double close, circular lifetime, or static-initialization issue was found.
- Configuration: No new public configuration is added. Existing no-cache/reload Paimon behavior makes the generation crossing reachable; REST vended credentials intentionally keep
latestSnapshotCachewhile disablingtableCache, making the Iceberg failure reachable. Existingpaimon.table-option.*precedence has a separate equal-value provenance gap already covered byr4004077352. - Compatibility: No persisted format, SPI symbol, EditLog, or FE-BE wire change is introduced. UUID-less format-V1 Iceberg metadata is an explicitly supported compatibility path, and the new exact-filename identity fallback breaks valid same-table commits there. No additional rolling-upgrade issue was found.
- Parallel paths: Latest, empty, reader-only OPTIONS, explicit selectors, branches, supported system wrappers, RANGE/LIST/UNPARTITIONED materialization, synchronous/streaming/count scans, native dictionaries, JNI serialization, and metadata tables were checked. No additional distinct issue survived; the known explicit Paimon historical-memo and Iceberg partition-view/spec points are already fenced by existing discussions.
- Conditions and error handling: The Iceberg mismatch error includes the table and retry guidance, but its condition is over-broad for ordinary UUID-less commits. Paimon numeric-ID/equality checks cannot prove physical identity. No separate swallowed status/exception or speculative defensive branch was found.
- Test coverage: The added unit/regression tests broadly cover cache hits, empty-to-first-append, recreation, schema rename, partition arity/type, synthesized versus generated columns, authentication, fallback/branch behavior, native/JNI paths, and timestamp precision. They miss recreation between one statement's pin and later scan-handle resolution, and an ordinary UUID-less M1-to-M2 commit under warm-pin/fresh-table caching.
- Test results/output: Query labels and expected regression output were checked statically. Per the review contract, I did not run builds or tests; author/CI-reported results are not independent verification.
- Observability: Existing errors carry useful table/retry context. Paimon generation mixing has no fail-loud signal, while the Iceberg signal fires on valid evolution; fixing those conditions is the actionable need. No separate logging or metrics gap was substantiated.
- Transactions/persistence: The changed code is read-metadata/planning logic and adds no Doris transaction, journal, failover, or persistent-state behavior.
- Data writes: No Doris data-write path is modified. Existing Iceberg writer identity ancestry was inspected as evidence for valid V1 evolution; no separate write atomicity issue was found.
- FE-BE variables: No new transmitted variable or scattered protocol send path is added; snapshot identity/schema/spec properties remain FE-internal. Native/JNI consumers were included in the consistency trace.
- Performance, memory, and BE nullability: This is FE Java code; no BE allocator/nullability issue applies. Schema/identity work is bounded to statement/cache operations. Apart from the recurring valid-query retry and the already-reported lost reader policy, no material CPU or memory regression was found.
- Other data correctness: Paimon timestamp precision 0-6 preserves the visible wall clock; precision 7-9 pushdown fails closed while Doris retains residual evaluation. The synthesized-column exemption remains narrow enough that physical/generated skew fails loudly. No further distinct issue was found.
User focus: No additional user-provided focus was supplied; the full authoritative diff was reviewed.
Completion: Complete after three convergence rounds. All candidates were accepted, dismissed with code evidence, or duplicate-fenced; this review submits the two remaining distinct findings.
|
run buildall |
|
/review |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
TPC-H: Total hot run time: 16649 ms |
TPC-DS: Total hot run time: 81073 ms |
ClickBench: Total hot run time: 14.5 s |
There was a problem hiding this comment.
Requesting changes after a complete static review of exact head 499a96e8c1171abf80d30fbc76d407b2a5a7d96a and all 22 changed files. The bounded review converged in round 3: every candidate is resolved, and the final set is two P1 consistency defects plus one P2 regression-oracle gap, detailed inline.
Critical-checkpoint conclusions:
- Goal and proof: The patch largely retains latest Paimon/Iceberg schemas with their data coordinates through metadata and scan consumers, but the goal is incomplete: the Paimon fallback child's data generation is not fenced, and Iceberg query-schema DDL can combine a pinned schema with a live sort order. The later reader-option behavior also lacks repository-standard generated regression output.
- Scope and focus: The implementation is otherwise focused on connector schema capture/validation, reconstruction, generic MVCC consumers, and timestamp predicate safety. I reviewed the complete authoritative diff and relevant callers/consumers. The user-focus file supplied no additional focus beyond the full PR.
- Concurrency: The accepted failures are reproducible with sequential operations; no race is required. Statement maps use immutable values/concurrent containers and connector caches retain their existing synchronization. I found no new lock-order, blocking-under-lock, unsafe-publication, or deadlock defect.
- Lifecycle/static initialization: Statement-owned pins and borrowed/tracked tables have bounded lifetimes and existing teardown. No circular ownership, resource leak, or static-initialization issue was found.
- Configuration: No configuration item is added. Existing positive Iceberg latest-cache TTL plus the REST-vended fresh-table path is the concrete trigger for the Iceberg issue; configuration parsing/dynamic-update behavior is unchanged.
- Compatibility/persistence: No public SPI, function symbol, thrift/protocol field, EditLog/image record, or storage format changes. I found no separate rolling-upgrade, replay, or persisted-state issue.
- Parallel paths and conditions: I checked latest/empty/time-travel pins, Paimon OPTIONS/branches/system/fallback/privilege/native/JNI paths, Iceberg direct/HMS/UUID-less/partition/handle/write paths, Hive/Hudi generic behavior, and
/api/query_schema. Identity mismatches otherwise fail loud; the two uncovered consistency arms are the inline P1s. - Tests and results: Changed unit/regression tests cover most schema evolution, recreation, authentication/TCCL, option precedence, partition, and generated/synthesized-column paths. Missing cases are fallback-only same-schema recreation, Iceberg sort evolution through query-schema DDL, and generated
.outentries for the later five deterministic reader-option checks. The existing five.outlabels match their queries. Per this review assignment, I ran no build or test, so execution claims are not independently verified. - Observability/error handling: Generation mismatch errors include retry guidance, and no additional metric is warranted for statement-scoped planning. The Iceberg defect silently emits inconsistent DDL, so it needs a correctness fix and regression rather than logging alone. No other error-propagation issue survived review.
- Transactions, persistence, and data writes: These changes are read/planning-side; they add no transaction, journal, or data-write atomicity path.
- FE/BE variables: No new session/system variable or thrift field crosses FE/BE. Existing connector snapshot properties/table options carry the bound state through reconstruction.
- Performance: Added schema/digest work is bounded per statement/table and memoized where designed. I found no unbounded loop, per-row work, or distinct hot-path regression.
- Other correctness: Paimon timestamp precision 0-6 remains exactly pushable and precision above 6 remains residual across comparison operators. No additional distinct correctness issue remained after duplicate fencing and final missed-area sweeps.
Review status: complete static review at the exact live head; changes requested for the three inline issues.
### What problem does this PR solve? Related PR: apache#67904 Problem Summary: Latest-schema retention was implicitly enabled by a schema ID, changing ordinary metadata publication for unrelated connectors. Make retention an explicit Paimon opt-in and remove the collateral Iceberg/Hive changes. Keep schema and physical coordinates in one immutable statement pin, including the fallback branch data fence, so reloads cannot silently select replacement data. Preserve upstream snapshot-aware partition listing when integrating master. ### Release note Preserve Paimon statement schemas and fallback data coordinates. The connector plugin API advances to 8.0 for explicit latest-schema publication. ### Check List (For Author) - Test: Related FE/connector suites, full SPI suite, FE build, standalone FE Checkstyle, and Native/JNI regression queries. - Behavior changed: Yes; only explicitly opted-in latest pins publish retained schemas, and changed fallback generations are rejected. - Does this need documentation: No; existing Paimon query semantics are restored.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review at exact head 31945b11ac196c3c31041cac0e09d13ba00d8390.
Status: capped/incomplete. The final allowed convergence round materially broadened the second finding from the equal-ID shortcut to both restoration branches, so the contract permits no fourth round.
Two distinct P1 issues remain; see the inline comments.
Checkpoints: the Paimon-only, default-false retainSchema scope and connector API-major/surface updates are consistent; non-Paimon latest behavior, storage/EditLog/thrift formats, and runtime configuration are unchanged. Statement pin map ownership, lifecycle, authentication/TCCL, branch/fallback/system routing, NTZ residual handling, and bounded metadata-read cost were traced without another distinct issue. Existing raw/rendered discussions were applied as a hard duplicate fence.
Static inspection only as required: no build, unit test, or regression test was run, and author/CI test claims were not independently executed.
### What problem does this PR solve? Related PR: apache#67904 Problem Summary: Schema restoration strictly reparsed properties accepted by catalog replay, making ordinary scans fail on ignored legacy reader options. It could also combine a replacement table's live schema with a retained old table object after DROP/CREATE. Preserve compatible option parsing and validate the retained schema against its own persisted version before either restoration branch, excluding legitimate dynamic option overlays from that comparison. ### Release note Keep replayed Paimon catalogs queryable and reject stale table objects during statement schema restoration. ### Check List (For Author) - Test: Four failing regressions reproduced before the fix; 598 Paimon tests (one existing connectivity test skipped), plugin packaging, and FE Checkstyle. - Behavior changed: Yes; restored scans retain replay compatibility and reject retained objects from a different physical schema generation. - Does this need documentation: No; preserves existing catalog/query contracts.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review conclusion: I found no new actionable issue on the current head after treating all existing inline threads and raw review comments as hard duplicate fences. I would approve this head based on the static evidence reviewed here.
Critical checkpoint conclusions:
- Goal and proof: the change now binds Paimon's latest schema independently of its data snapshot, carries physical schema/snapshot coordinates through the statement, validates reusable IDs before scan reconstruction, and preserves that schema through FE materialization and native/JNI planning. The added FE/SPI tests and external regression cover schema-only ALTER, empty/first-append and DROP/CREATE races, fallback branches, system/privilege wrappers, option precedence, and timestamp precision. Test execution reported in the PR/threads is author/CI evidence; I did not run builds or tests under this review contract.
- Focus and smallness: although the diff is necessarily cross-layer, each public SPI/FE change is tied to the Paimon opt-in. The connector-local digest/restoration helpers keep generation validation and decorator reconstruction centralized; I found no unrelated production behavior change.
- Concurrency: the relevant concurrency is external Paimon ALTER/commit/DROP/CREATE racing FE binding and planning.
statementPinsis a per-metadataConcurrentHashMap, its immutable values are computed once per logical base/system/branch handle, and expensive catalog/file reads occur outside Doris locks. No new lock ordering or deadlock surface is introduced; generation changes fail closed at restoration. - Lifecycle: one metadata instance is memoized for one statement, so its pins expire with the statement; INSERT scope replacement rehydrates from carried coordinates. Catalog-wide snapshot/schema/partition caches retain their pre-existing lifecycle. No static-initialization, circular-reference, or resource-release issue was found.
- Configuration: no new configuration key is added. Existing Paimon catalog and relation reader options retain relation-over-catalog-over-physical precedence, replay filters incompatible historical values, and validation occurs at the effective planning boundary.
- Compatibility:
ConnectorMvccSnapshot.retainSchemadefaults false, only Paimon latest pins opt in, the frozen SPI surface is updated, and the connector API major is bumped to 8 so incompatible external plugins fail closed. No storage format, EditLog, or new Thrift field is introduced. - Parallel paths: I traced plain latest reads, selector-free OPTIONS, explicit snapshot/tag/time, incremental and branch reads, empty tables, system tables, partition materialization, native planning, JNI/backend serialization, and transient-table reload. Other connectors keep the default-off schema-retention behavior.
- Conditions and errors: the new retain-schema, empty-fence, wrapper, synthesized-column, and timestamp-precision branches are narrowly gated and documented. Physical/generated schema misses and generation/digest mismatches remain fail-loud with retry context; unsupported decorators fail loudly rather than dropping policy.
- Data correctness: schema and data fences are validated before the executable scan, fallback children retain independent coordinates, and sub-microsecond source timestamps remain residual while precisions up to microseconds preserve the full literal. No changed-line wrong-row path survived the final trace.
- Test coverage and results: changed tests include positive and negative unit cases plus ordered regression queries for plain/options/branch and native/JNI behavior. The checked
.outlabels align with the query labels and deterministic ordering requirements. I did not independently regenerate the result file. - Observability: the retry error identifies a changed Paimon table generation and existing planning/authentication logs retain table context. This statement-local correctness fence does not require a new long-lived metric.
- Persistence, writes, and crash behavior: this is a read-planning/SPI change; it adds no Doris transaction, data-write, EditLog, or persistent-state path, so master failover and write atomicity are not implicated.
- FE/BE variables: no new FE-to-BE protocol field is added. The already-serialized Paimon table and schema-evolution dictionary are rebuilt from the same bound source for native and JNI paths.
- Performance and memory: the opted-in path adds one eager mapped-schema materialization per table/statement; statement pins and existing catalog caches bound repeated work. Digest maps are statement-scoped, and no unbounded collection, hot-loop I/O, BE allocator, nullable-column, or obvious asymptotic regression was found.
- Other issues and review focus: no additional user focus was supplied. A final missed-area sweep of all 22 changed files, existing comments, and the shared risk ledger found no distinct issue left to post.
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
|
PR approved by at least one committer and no changes requested. |
What problem does this PR solve?
Paimon schema-only ALTER operations can advance the schema without creating a data snapshot. Reusing a data fence as a schema selector can rewind the analyzed schema, and timestamp predicate conversion can lose fractional precision and prune matching rows.
This PR keeps a statement's Paimon schema independent of its data fence. A single immutable statement pin carries the schema and physical schema/snapshot coordinates, including fallback branch-local data coordinates. Metadata reloads and INSERT replanning either reuse that generation or reject a changed physical table. Reader options retain catalog and relation override precedence and replay compatibility while preserving authentication and table decorators. Retained table objects are checked against their own physical schema version before restoring a pin or deriving option overrides. NTZ predicates preserve microseconds; comparisons requiring nanosecond precision remain Doris residual predicates.
Scope and compatibility
Latest-schema publication in the generic FE bridge is an explicit connector opt-in, enabled by Paimon. A positive schema ID alone does not change ordinary schema publication. Iceberg and Hive retain their existing behavior. Existing explicit time-travel cache policies and historical Iceberg partition-spec isolation are outside this PR.
The typed opt-in changes the connector plugin contract, so its API version advances from 7.0 to 8.0. Connector plugins must be rebuilt for the matching API; the public surface baseline and compatibility tests cover this contract.
Release note
Fix Paimon schema consistency after schema-only changes and preserve timestamp predicate precision. Validate retained statement generations across metadata reloads and fallback branch recreation.
Test
install.test_paimon_schema_only_snapshot_precisionpassed. The current follow-up changes only the Paimon module; the regression output was generated by the harness.Check List (For Reviewer who merge this PR)